{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6-0_o3DxsFGi"
   },
   "source": [
    "# Google Memorystore for Redis\n",
    "\n",
    "> [Google Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview) is a fully-managed service that is powered by the Redis in-memory data store to build application caches that provide sub-millisecond data access. Extend your database application to build AI-powered experiences leveraging Memorystore for Redis's Langchain integrations.\n",
    "\n",
    "This notebook goes over how to use [Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview) to [save, load and delete langchain documents](https://python.langchain.com/docs/modules/data_connection/document_loaders/) with `MemorystoreDocumentLoader` and `MemorystoreDocumentSaver`.\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/googleapis/langchain-google-memorystore-redis-python/blob/main/docs/document_loader.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Before You Begin\n",
    "\n",
    "To run this notebook, you will need to do the following:\n",
    "\n",
    "* [Create a Google Cloud Project](https://developers.google.com/workspace/guides/create-project)\n",
    "* [Create a Memorystore for Redis instance](https://cloud.google.com/memorystore/docs/redis/create-instance-console). Ensure that the version is greater than or equal to 5.0.\n",
    "\n",
    "After confirmed access to database in the runtime environment of this notebook, filling the following values and run the cell before running example scripts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# @markdown Please specify an endpoint associated with the instance and a key prefix for demo purpose.\n",
    "ENDPOINT = \"redis://127.0.0.1:6379\"  # @param {type:\"string\"}\n",
    "KEY_PREFIX = \"doc:\"  # @param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🦜🔗 Library Installation\n",
    "\n",
    "The integration lives in its own `langchain-google-memorystore-redis` package, so we need to install it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -upgrade --quiet langchain-google-memorystore-redis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Colab only**: Uncomment the following cell to restart the kernel or use the button to restart the kernel. For Vertex AI Workbench you can restart the terminal using the button on top."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Automatically restart kernel after installs so that your environment can access the new packages\n",
    "# import IPython\n",
    "\n",
    "# app = IPython.Application.instance()\n",
    "# app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ☁ Set Your Google Cloud Project\n",
    "Set your Google Cloud project so that you can leverage Google Cloud resources within this notebook.\n",
    "\n",
    "If you don't know your project ID, try the following:\n",
    "\n",
    "* Run `gcloud config list`.\n",
    "* Run `gcloud projects list`.\n",
    "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# @markdown Please fill in the value below with your Google Cloud project ID and then run the cell.\n",
    "\n",
    "PROJECT_ID = \"my-project-id\"  # @param {type:\"string\"}\n",
    "\n",
    "# Set the project id\n",
    "!gcloud config set project {PROJECT_ID}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🔐 Authentication\n",
    "\n",
    "Authenticate to Google Cloud as the IAM user logged into this notebook in order to access your Google Cloud Project.\n",
    "\n",
    "- If you are using Colab to run this notebook, use the cell below and continue.\n",
    "- If you are using Vertex AI Workbench, check out the setup instructions [here](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/setup-env)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.colab import auth\n",
    "\n",
    "auth.authenticate_user()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2L7kMu__sFGl"
   },
   "source": [
    "## Basic Usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save documents\n",
    "\n",
    "Save langchain documents with `MemorystoreDocumentSaver.add_documents(<documents>)`. To initialize `MemorystoreDocumentSaver` class you need to provide 2 things:\n",
    "\n",
    "1. `client` - A `redis.Redis` client object.\n",
    "1. `key_prefix` - A prefix for the keys to store Documents in Redis.\n",
    "\n",
    "The Documents will be stored into randomly generated keys with the specified prefix of `key_prefix`. Alternatively, you can designate the suffixes of the keys by specifying `ids` in the `add_documents` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import redis\n",
    "from langchain_core.documents.base import Document\n",
    "from langchain_google_memorystore_redis import MemorystoreDocumentSaver\n",
    "\n",
    "test_docs = [\n",
    "    Document(\n",
    "        page_content=\"Apple Granny Smith 150 0.99 1\",\n",
    "        metadata={\"fruit_id\": 1},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"Banana Cavendish 200 0.59 0\",\n",
    "        metadata={\"fruit_id\": 2},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"Orange Navel 80 1.29 1\",\n",
    "        metadata={\"fruit_id\": 3},\n",
    "    ),\n",
    "]\n",
    "doc_ids = [f\"{i}\" for i in range(len(test_docs))]\n",
    "\n",
    "redis_client = redis.from_url(ENDPOINT)\n",
    "saver = MemorystoreDocumentSaver(\n",
    "    client=redis_client,\n",
    "    key_prefix=KEY_PREFIX,\n",
    "    content_field=\"page_content\",\n",
    ")\n",
    "saver.add_documents(test_docs, ids=doc_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "A2fT1iEhsFGl"
   },
   "source": [
    "### Load documents\n",
    "\n",
    "Initialize a loader that loads all documents stored in the Memorystore for Redis instance with a specific prefix.\n",
    "\n",
    "Load langchain documents with `MemorystoreDocumentLoader.load()` or `MemorystoreDocumentLoader.lazy_load()`. `lazy_load` returns a generator that only queries database during the iteration. To initialize `MemorystoreDocumentLoader` class you need to provide:\n",
    "\n",
    "1. `client` - A `redis.Redis` client object.\n",
    "1. `key_prefix` - A prefix for the keys to store Documents in Redis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "YEDKWR6asFGl"
   },
   "outputs": [],
   "source": [
    "import redis\n",
    "from langchain_google_memorystore_redis import MemorystoreDocumentLoader\n",
    "\n",
    "redis_client = redis.from_url(ENDPOINT)\n",
    "loader = MemorystoreDocumentLoader(\n",
    "    client=redis_client,\n",
    "    key_prefix=KEY_PREFIX,\n",
    "    content_fields=set([\"page_content\"]),\n",
    ")\n",
    "for doc in loader.lazy_load():\n",
    "    print(\"Loaded documents:\", doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete documents\n",
    "\n",
    "Delete all of keys with the specified prefix in the Memorystore for Redis instance with `MemorystoreDocumentSaver.delete()`. You can also specify the suffixes of the keys if you know."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = loader.load()\n",
    "print(\"Documents before delete:\", docs)\n",
    "\n",
    "saver.delete(ids=[0])\n",
    "print(\"Documents after delete:\", loader.load())\n",
    "\n",
    "saver.delete()\n",
    "print(\"Documents after delete all:\", loader.load())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced Usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "02xxvmzTsFGm"
   },
   "source": [
    "### Customize Document Page Content & Metadata\n",
    "\n",
    "When initializing a loader with more than 1 content field, the `page_content` of the loaded Documents will contain a JSON-encoded string with top level fields equal to the specified fields in `content_fields`.\n",
    "\n",
    "If the `metadata_fields` are specified, the `metadata` field of the loaded Documents will only have the top level fields equal to the specified `metadata_fields`. If any of the values of the metadata fields is stored as a JSON-encoded string, it will be decoded prior to being loaded to the metadata fields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BvS3UFsysFGm"
   },
   "outputs": [],
   "source": [
    "loader = MemorystoreDocumentLoader(\n",
    "    client=redis_client,\n",
    "    key_prefix=KEY_PREFIX,\n",
    "    content_fields=set([\"content_field_1\", \"content_field_2\"]),\n",
    "    metadata_fields=set([\"title\", \"author\"]),\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": [
    {
     "file_id": "1kuFhDfyzOdzS1apxQ--1efXB1pJ79yVY",
     "timestamp": 1708033015250
    }
   ]
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
